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TESTING OF EMBEDDED SYSTEMS 

FIELD OF INVENTION 

The present invention relates to the field of embedded or distributed systems. 
5 In one form, the present Invention relates to a method and apparatus for the testing 
of embedded devices. It will be convenient to hereinafter describe the invention In 
relation to the use of a method and apparatus employing, in part, CASE (Computer 
Aided Software Engineering) tools to model the behaviour of an embedded device or 
system refened to herein as a Device Under Test (DUT). however It should be 
10 appreciated that the present Invention is not limited to that use only. 
BACKGROUND ART 

Throughout this specification the use of the word 'inventor^ in singular fonn 
may be taken as reference to one (singular) or all (plural) inventors of the present 
Invention* 

15 The inventor has identified and considered the following technologies. 

In general, an embedded system is a system within a larger system. 
Embedded systems may be Implemented on a single integrated circuit or as scaled- 
down software code. An embedded system typically has a specialized function with 
programs stored on ROM. Examples of embedded systems are chips that monitor 

20 automobile functions, comprising engine controls, antilock brakes, air bags, active 
suspension systems; industrial process devices* for example comprising robotic 
systems in production engineering applications or medical applications; 
environmental systems, comprising poltutton monitoring devices etc; entertainment 
systems and; property protection systems, comprising for example security devices, 

25 fire prevention or smoke detection devices and combinattons thereof. Ordinarily, 
everything needed for those functions is custom designed Into specific chips and, 
there may be no external operating system required. Many embedded systems are 
produced in the range of tens of thousands to millions of units. Reducing costs In all 
aspects of design to actual production, installation and maintenance Is therefore a 

30 major concern. There may be many different CPU architectures used in embedded 
designs. This contrasts with the desktop computer market, which Is limited to only a 
small number of competing archttectuies, such as, Intei^s x86, and the 
Apple™/Motorola™/IBM'™ PowerPC™. A common configuration for embedded 

1 
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^teims i8 the "system on a ohip''^ an appti€:atipn-specifiQ integrated circuit It is also 
notable that user Interlaces for embedded systems vary to a large extent. The 
resultant complexity of most embedded systems requires that generally they are 
developed In teams by several developers at the same time using different computer 
5 pliatforms- Furthermore, high perfomnance micro-controllerB which are now available 
may provide developers with highly complex software solutions for embedded 
system design. Managing the complexity of software components and their intern- 
workings has therefore also become a meyor concern. 

Rational™ Robot is a PC application designed for the design and execution of 
10 ^ tests on PC applications. Therefore, In an attempt to pro\dde a test environment for 
embedded systems by use of Rational^^ RotK>t, a Data Driven Engine (DDE) is 
required to deal with the complexi^ of generating tests within a PC environment and 
translating those tests into a syntax that may be interpreted by additional interface 
hardware. 

15 Vlsio^ is an application where a message sequence chart (msc) representing 

a test case or sequence may be constructed and then via interfacing Visual Basic 
code representing the Vlsio™ msc to Labview™, a graphical programming language 
tooL The Labview™ functions may then interpret this code and execute the test on 
a DUT, It is to i>e noted that LabvJew^'*^ being the programming environment, still 

20 needs to interface physically and electrically to the DUT via specific I/O hardware. 
There is also an Issue of having to reconstaict from scratch the entire msc every 
time there is a change to the underiying test sequence requirements. 

A product from Lucent™ known as UBET employs an enhancement of msc's, 
namely, HIMSC (l-|ierarchicat Message Sequence Charts). The llmitatbns of the 

25 Visual Basic scripts are removed in this implementation, however, the Inventor has 
identified that there still remains the issue of reconsbruoting from scratch the MSC's 
and HMSC's with every iteration In requirements or change In requirements of a test 
sequence for embedded systents. 

Generally, "black boyC test environments do not exist for embedded systems, 

30 in particular, systems that are not built on or do not employ Industry standatxl 
communications protocols, in a niche market, such as is filled by the fire and smoke 
alarm products developed by Vision Fire & Security Pty Ltd and marketed under 
trade marks such as VESDA®. the communications protocols are developed in 
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house and are not serviced by m^'or Industry developers of automation systems, for 
example, such a$ TTCN2 and TTCN3- Thus no means exists to leverage off the 
activHy of cooperating Industries and user groups. 

In general, test automation environments such as Rational" Robot are 
5 contemporaneous with other automation environments such as WinRunner™ and 
other PC application and GUI focused applications. The inventor has identified that 
these environments are designed In a manner, which may not communicate with, 
stimulate or control embedded systems. The inventor has further identified that 
attempts to retrospectively adapt these environments through the addition of 
10 interpreters or hanlware interfaces necessitates very large efforts in terms of 
scripting and definition of key wofds and phrases. Moreover, the Inventor has 
Identified that test designing within these environments requires repetitive manual 
labour. Overall, the inventor has identified use of these environments to be time 
consuming, difficult to maintain and labour intensive in generation and updating and. 
IS unable to adequately deal with any changes in requirements and project scope. 

The inventor has iiirther Identified that, under such environments, bridging the 
divide from formally defined tests to physically interface to a product that is 
controlled via embedded software and automatically execute the tests introduces 
difficulty given that very few user accessible interfaces are generally available for 
20 stimulus and reading of responses from embedded DUTs. 

Any discussion of documents, devices, acts or knowledge in this specification 
is included to explain the context of the invention. It should not be taken as an 
admission tliat any of the material forms a part of the prior art base or the common 
general knowledge in the relevant art in Australia or elsewhere on or before the 
25 priority date of the disckssure herein. 

An object of the present invention is to automatically execute functional tests 
for erhbedded systems at a system-wMe or black t>ox level In a manner that Is 
responsive to changing test requirements. 

A further object of the present invention is to alleviate art least one 
30 disadvantage associated with the prior arL 

The scope of applicability of the present invention will l)ecome apparent from 
the detailed description given hereinafter. However, it should be understood that the 
detailed description and specific examples* while indicating preferred embodiments 

3 
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of the invention, are given by way of illustration only, since various changes and 
modifications v\dthin the spirit and $cope of the invention will become apparent to 
those skilled in the art from this detailed description. 
BRIEF DESCRIPTION OF THE DRAWINGS 
5 Further disclosure, objects, advantages and aspects of the present invention 

may be better understood by those skilled in the relevant art by reference to the 
follovi^ng description of preferred embodiments taken (n conjunction with the 
accompanying drawings, which are given byway of illustration only, and thus are not 
limiting to the scope of the present Invention, and in which: 
10 Figure 1 is a block diagram schematicaiiy iliustraang an automated model 

architecture and test environment in accordance with an embodiment of the present 
invention; 

Figure 2 is a block diagnam schematically illustrating arohitectuFe of a generic 
test execution engine in accordance with an embodiment of the present invention; 
15 Figure 3a illustrates a file showing the results of a comparison t>etween 

modelled device behaviour and actual device behaviour wherein a device has 
passed an automated test in accordance with an embodiment of the present 
invention; 

Figure 3b illustrates a file showing the results of a comparison between 
20 modelled device behaviour and actual devt(^ behaviour wherein a device has failed 
an automated test In accordance with an emtx^diment of the present Invention; 

Figure 3c illustrates a file showing the results of a masked comparison 
between modelled device l>ehavlour and actual device behaviour wherein a device 
has passed an automated test in accordance with an embodiment of the present 
25 invention; 

DETAILED DESCRIPTION 

By way of background, a numl>er of terms are here defined. 

DUT Device Under Test A taiiget embedded device or, an embedded 
system comprising one or more embedded devices tiiat is the subject of tests* 
30 Furthermore* a DUT may comprise one or more embedded systems being the 
subject of tests* 

(Model An approximation of selected aspects of the behavioural 
characteristics of a real-worid embedded DUT and real world influences on the 

4 
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embedded DUT. A model may use Architectural Level Descriptions (ALD) or 
Definitions, State Tran^ction Definitions. Extended Message Sequence Charts 
(EMSC) and Data Type Definitions to provide an abstract representation of the 
physical interface, real world environmental influences and constraints upon an 

5 embedded DUT» 

Output Port An output port is used In a model to represent a physical output 
on the DUT being modelled. The physical outputs of a DUT may comprise LEO'$» 
r^lay contacts, Open Collector, Wet or Dry Outputs, Communicatione Ports, for 
example, RS-232 employing AOCP etx>, and Ethernet Ports. 

10 TVF Teat Vector Re, A formatted asdi text file containing Test Vectors 

generated based on definitions contained in a test configuration file (TCF). There 
may be two versions of a TVF utilised in embodiments of the present 'invention, an 
input test vector only TVF and a inputf output test vector (response) TVF created 
when the input only test vector TVF is passed through the model. 

IS Test Vector An ordered set of input, output, or Input and output values 

defining input and/or output information. For example a test vector may comprise a 
line within a TVF representing a command set of inputs or input/outputs. 

Test Vector Pair A Test Vector pair is the same as a test vector, however, 
the outputs represented are valid at that point In time indicated by the value of a 

20 predefined timing reference provided in accordance with embodiments of the 
present invention. The predefined timing reference may be provided by way of a 
Timing Port- 

Cloclc Pulse A clocic pulse is an arbitrary but fixed timing period employed by 
a device model to synchronise activity within the model resulting in propagation of 
25 variables and transition firing. The period of time within the model during which 
Inpute are sensed, calculations computed and outputs generated. 

Conflg Port A port ttiat is used by a test engineer/test designer to identify for 
the automated execution engine of an embodiment of the present invention that the 
parameter associated with the port is a configuration parameter and should be 
30 applied before any stimulus is applied to the input ports. 

GEE A Generic Execution Engine in accordance with an embcxliment of the 
present invention. 



5 
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Execution Port A port that Is used by the GEE to initiate and maintain 
communication with a DUT Speciflc InterfacG. 

TCF Test configuration file, A set of instnjctions that define the 
configuration parameters, stimuli parameters (pattems), the permutation and 

5 combinations of ail input ports with port Input patteme, test repetition rates and test 
group repetition rates, and the definition of the explicit test cases under the 
preceding settings which re^^ults in the generaQon of input (etimuli) test vectors 
associated with input only test vector sets, which may be In the fomn of test vector 
files (TVF's). The TCP therefore effecfiveiy comprises the test sequence rules. The 

10 Autofocus™ modelling tool pn>vides for the creafion of GCF's {Group Configuration 
Flies), which may i>e viewed as a test sequence configuration facility. However, the 
TIG (Test Input Generator}^ used to create GCPs in Autofbcus^, is unable to 
produce configuration files for floating models using floating point arithmetic The 
TCF's of the present Invention provide for what is Icnown in graph theory as a 

15 "guided tour^. That is the State Transition Diagram of a model is considered as a 
graph through which the execution of test sequences is guided in such a manner as 
to directly influence and control the mode and manner in which testing pn^gresses. 
As a result, the present invention may be in control of what gets tested, how it gets 
tested and to what degree (coverage) it is tested. This features is not a natural 

20 consequence Of the GCF from an Autofocus^ TIG, where a GCF will result in a 
random unguided mode of execution. 

Input Port An input port is used in a DUT, and in parQcular a DUT model to 
represent a physical input on the DUT being modelled. 

Timing Port A port that is used by the tj^st designer to indicate to the 

25 automated execution engine at what time. In accordance with a predefined timing 
reference, to sample the outpute of the DUT (modelled or actual) for matches of 
stabilised output vectors with corresponding input vectors. Furthermore, in the case 
of obtaining outputs from an actual DUT when provided with the input vector (which 
has a corresponding model generated output vector) the DUT output is sampled for 

30 the output vector or response* which is then compared against the model generated 
output vector of the same input vector. The DUT response may therefore t>e 
determined for its equivalence to that of the model for the same stimuli. The Timing 
Port is a port that doesnt necessarily exist on the DUT. The execution engine 

6 
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executes a wait period determined by the predefined timing reference indicated on 
the timing port directly after applying stimulus to the input port. 

In one aspect, a preferred embodiment of the present invention provides In an 
embedded DUT testing system comprising apparatus adapted to compare actual 
S DUT input^output vector pairs with modelled DUT input/output vector pairs, a logical 
connection port adapted to indicate a predefflned timing referance for determining a 
point in time at which to sample an output vector as the corresponding output vector 
in an input/output vector pair. The sampled output vector may be a modelled or 
actual output vector of one or more of: 
10 a amolce detector. 

a fire deteoton 

a securtty device; 

a medical device; 

a biological tissue processing device; 
IS an industrial process device. 

In another aspect, a preferred emt)odiment of the present invention provides 
. a method of testing at least one embedded DUT comprising the steps of: 

a) determining a test configuration parameter set comprising predefined DUT 
test sequence rules; 

20 b) determining a first data set comprising input test vectors based on the test 

configuration parameter sat; 

c) processing the first data set in a DUT model to determine output test 
veotoi? wherein the output test vectors comprise DUT model generated responses 
to the Input test vectors; 
25 d) processing the first data set and the output test vectors to determine a 

second data set comprieing pairs of stabilised input and output test vectors; 

e) communicating the stabilised input test vectors to at least one DUT via a 
DUT independent interface so that the at least one DUT Is stimulated by the 
stabilised input test vectors to pmduce DUT output vectors; 
30 f) determining a third data set comprising the stabilised input vectors and 

corresponding DUT output vectors; 

g) comparing the third data set with the second data set to determine a 
comparison of actual behaviour to modelled t>ehaviour of the at least one DUT. 

7 
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Step d), above, may further comprise the steps of. 

h) parsing the output test vectors with the first data set in accordance wHh a 
. piedefined timing reference in which the predefined timing reference determines a 
point In time to sample an output test vector as a stabilised output test vector; 
5 I) matching each stabilised output teat vector to a corresponding input test 

vector to form pairs of stabilised input and output test vectors. 

The predefined timing reference may be derived from a logical connection 
port as described at>ove. Further, the predefined timing nsforence may comprise 
one of: 

10 one delay period common to all input test vectors, and; 

a prodatermined delay period for each input test vector. 

The DUT independent interface may comprise an interprocess 
communication protoool utitising one of: 

TCP/IP, and; 
IS Active-X. 

The DUT model may comprise one or more of the following abstraction 
techniciues: 

architectural level descriptions; 

data type definitions; 
20 state transition diagrams; 

extended Message Sequence Charts. 

The data sets described above may comprise test vector fomnatted flies. 
The test configuration parameter set described above may comprise a test 
parameter configuFation file. 
25 . The DUT may comprise one or more of: 

a smoke or fire detector; 
a security device; 
a medical device; 

a biological tissue processing device; 
30 an industrial process device. 

In yet another aspect^ a preferred embodiment of ttie present invention 
pn>vldes apparatus adapted to test at least one em]:>edded device (DUT), said 
apparatus comprfsing: 

8 



COMS ID No: SSMI-00819663 Received by IP Australia: Time <H:m) 16:36 Date (Y-M-d) 2004-07-07 * 



7- 7-04jl6!23 IDawlBS ColM$on Caue 



secure fax IP AUSTJ61 3 92542808 



#15/ 3 6 



processor meand adapted to operate in accordance with a predetermined 
InstmcUon set, 

, said apparatus, in conjunction with said instouction set, being adapted to 
perform the method steps as described herein, 
5 In still another aspect, a prefened embodiment of the present invention 

provides a generic test execution engine for testing at least one embedded DUT 
comprising: 

a test vector interface comprising parsing means for parsing modelled output 
test vectors with predefined input test vectors fn accordance with a predefined timing 
10 reierence to determine a test vector file comprising pairs of corresponding stabilised 
input and output test vectors; 

a DUT intertacB comprising addressing means for addressing, via data ports, . 
test vectors to respective DUFs In accortiance with an address Identtfier within the 
test vector file, and communication means for receiving DUT output vectors in 
15 response to the test vectors; 

a test Insult interface comprising processing means for processing respective 
pairs of stabilised input and output test vectors and corresponding received DUT 
output vectors. As noted above, the predefined timing reference may be derived 
from a logical connection port as descril^ed herein. The predefined timing reference 
20 may determine a point in time to sample an output test vector In order to be 

determined as a stabilised output test vector for a corresponding input test vector. 

The generic test execution engine as described herein may be utilised for 
testing at least one DUT wherein the at least one DUT comprises one or more of: 
a smoke or fire detector. 
25 a security device; 

a medical device; 

a biological tissue preceding device; 
an Industrial process device. 

In a preferrsd embodiment of the present invention an embedded device 
30 testing system is provided comprising: 

a generic test execution engine as described herein; and, 
logging means operatively associated with the test result interface for 
recording an indication of a comparison between respective pairs of stabilised input 

9 
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and output test vectors and corresponding racalved OUT output vectors. The 
recoKJed Indication may comprise a formatted file comprising the following fields: 

DateTimeStamp; 

Evaluated Result; 

A Binary String repreoenting a logical comparison of each port value between 
respective tost and DUT output vectors; 
Port Identifier; 

Port value comprising one of matched port value and expected value/actual 

value. 

In still a further aspect, a preferred embodiment of the present invention 
provides a data Ibmiat for use in an embedded DUT testing system, the data format 
comprising an Input test vector field correspondingly paired with an output test vector 
field wherein: 

the Input test vector field comprises predefined input information; 

the output test vector field comprises stabilised output Infomfiation determined 
by $ampling output information, provided In response to con-esponding predefined 
Input information, at a point In time detemilned by a predefined timing reference. 
The predefined timing reference may be derived from the predefined input 
information. Further, the predefined input information may comprise command set 
input data and the pnedeflned output information may comprise command set output 
data. 

In yet a further aspect, a preferred embodiment of the present Invention 
provides an embedded DUT testing system data file comprising a data format as 
herein describedL 

In still another aspect, a prefenred embodiment of the present invention 
provides a data transmission packet fomiat for use in an embedded DUT testing 
system comprising a data format as herein described. 

In yet another aspect* a preferred embodiment of the present invention 
provides a computer program product including: 

a computer usable medium having computer readable program code and 
computer readable system code embodied on said medium for testing at least one 
embedded DUT within a data processing system, said computer program product 
comprising: 

10 
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computer readable code within $aid computer usable medium for performing 
the method steps of a method as herein described. 

In essence, the present invention stems from the realisation that determining 
stabilised output data by, for example, providing a predefined timing reference to 
S determine a point in time in which to sample or accept a given outpu(, of either a 
modelled DUT or an actual DUT in response to a given input, allows for a realistic 
evaluation of fhe behaviour (modelled or actual) of a DUT, which is con^rained to 
operate in a real environment Determining realistic or stabilised output allows 
. reliable test vectors to be determined much sooner than in the case of an ad hoc 
10 approach to determining the output data corresponding to a given input. The 
stabilised output data may then be combined with its conresponding input data to 
provide a data format with which to interfaoe, drive and subsequently test a plurality 
of DUT's. The plurality of DUT's may t>e muttipies of the same DUT or alternatively, 
multiple DUT's being different devices whether they provide the same function or 
13 not In a preferred embodimmt of the Invention, one or a plurality of the same DUT, 
eg a fire detection device* is/are subjected to the same or different tests. 

The present invention has been found to result in a number of advantages* 
such as: 

• Rapid prototyping of the functionaiity and behaviour specified for new 
20 products 

• Dynamic interaction with the DUT model results in earty detection of 
requirement defects 

• . Based on supplied user (test engineer) defined or handcrafted 
. configuration files containing test sequence rules* an almost unlimited number of 

2S tests and results can t>e automatically generated based on the DUT model 
behaviour 

• No reliance on or need for Data Driven Engines or similar as all 
necessary variable names are maintained from wittiin the model and linked to 
LabvIew™Vi's* 

30 • Automatic execution of the resultant executing test vector sets* which 

may be in the form of Execution Test Data RIes (ETDF's when employing the 
Autoft>cus™ modelling tool, described t>e]ow) via a graphical programming language 
interface to one or more I/O devices and subsequent reporting is automatic 

11 
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Parsing and checking for differences between nf»odel predicUons and 

actual la&ults is automatic 

• Increased requirement/featuro interaction (test sequence/mles) 

coverage 

5 • Increased beiiaviourai tests 

• Speed of test execution end reporting 

« Elimination of human interaction during automated test execution 

• improved repeatability in, for example, regression test execution 

f Faalitatlon of complete regression test execution of all generated tests 

10 at minimal expense 

Tiie inventor considers one major benefll or advantage of the Invention is the 

automation of all manual, time intensive, tedious and error prone test generation and 

execution tasks. 

With reference to Fig 1, an automated modelling and test environment 10 Is 

15 now generally described as it relates to a preifsnned embodiment of the prssent 
Invention. Dynamto. executable and configurable abstracted models 101 are 
created ftom which are derived responses corresponding to input stimuli 102. The 
models are predicated on specific test sequence rules 100 in the fonn of test 
coftflguraltoh parameter sets, which may be embodied in the fomn of a test 

20 configuratton file TCP as defined above. Generally speaking, the appropriate pairing 
by means of parsing 103 the Input stimuli to stabilised output response of a model in 
acconjanoe with an embodiment of the present invention provides. In a sense, a test 
oracle 108 for reference against actual DUT 105 behaviour. A test oracle 108 as 
such. In the context of this specification, provides a means by which the dynamic 

25 behaviour of an embedded device/system under test DUT 105 may be dynamtealiy 
interacted/interfaced 108 with and interrogated and from which a priori knowledge of 
potential responses to input sttmuti may be derived. The resultant sets or files of 
input to stabOised output may be retained In a "golden log" or oracle 108 as a 
prediction against which the actual responses 107 of the OUT 105 for the same 

30 stimuli 104 may be evaluated by a oomparfson 109 and from which a deviatton from 
expected behaviour based on the original test sequence niles/requlrements set of 
the actually Implemented devices may be observed. From a separate Ate that only 
has input or stimuli vectors a drive TVF fHe msfy be created 104 and used to drive an 
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interface environment 109 or harness that provides the etectricel and control 
interfaces to the device(s)/system(s) under test. DUT 105. By means of capturing 
the actual device under test responses to the input stimuli derived from the model as 
described above, an output file or log 107 of stimuli to response may be generated* 
5 This actual output file may then be compared 109 to the "golden file" of input to 
output derived fix>m the model, namely, the test oracle 108* Suitable scripts, for 
example, written in Peri or Labview™ elements may Implement the comparison, 
The differences identified by this comparison may be noted and Investigated for 
defects wlttiln: the Implemented DUT 105; defects In the requirements 100; or, 
10 defects in the test environment generally. 

The testing environment 10 of embodiments of the presertt invention rather 
than requiring manually hand crafWng test sequences based on Individual 
requirements in a very static sense which then need to be associated with a unique 
data driven engine or similar and automation of test thndugh a GUI based engine 
15 testing design Is instead based on dynamic behaviour expressed In a model 101 that 
may be interacted with. Hence test design may be a by-product of interacting with a 
model 101 that encapsulates the essence of the behaviour of the device or product 
in question DUT 106- By providing a model 101 with a series of inputs, the model 
produces outputs that are predictions of the actual behaviour of the actual DUT 106. 
20 Thus by appropriate application of input vectors in accordance with the present 
invention and generating tests, small to extremely large sets of tests can be quicidy 
generated. In a prefenred embodiment of the present Invention, It has been 
Identified that combinatorial generation of tests per se may be provided by 
commercial products such as Autofocus™. The present Invention limits, in a crude 
25 but effective way, the domain space explosion Inherent. In combinatorial generation 
through AutofQcaJs"^ by explicitly crafting the parameters (within a TCF) that 
represent the input domain carefully so as to minimise this domain explosion. It is a 
crude approach in as much as it may be reliant on the knowledge of the test 
designer. Other solutions to the domain state explosion caused by combinatorial 
30 effects have been postulated^ this includes application of CLP (Constraint Logic 
Programming) techniques, model proving tools such as SATO etc. However all 
these solutions so far reside very heavily in the research domain and have not yet 
made an effectual transition to industrial practice. Maintaining a record of tests is 

13 
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resolved by recognising that a mcKlel 101 may be used as the repository of the 
behaviour that gave rise to the tests generated. By altering the model 101 to keep 
pace wfth changing lequirements 100 and reusing the Input stimuli, all tests may be 
updated as required with almost no human interaction. In a stralghtfonvard 

5 approach, the model Is an abstraction that is moulded to represent the DUT 
behaviour In question. The GEE and DSr-s (Device Specific Interfaces) have 
knowledge of the terms and parameters and ports employed by the model. As long 
as the model does not change the implementation of these terms, parameters, ports 
etc. then there may be no issue. If changes are made In the model by the 

10 amendment of STD's, SSD's etc then the GEE end DSrs need to be amended to 
account for this change. This may be implemented, for example, in Labvlew™ By 
execution of tests 104 through, for ^mple a Labview'^'^ environment, with 
automaticdily based sets of stimuli and capturing the output results for automatic 
comparison with predicted output again eliminates or reduces dramatically human 

IS interaction. 

Given that models ans re-u$able then the reuse of model components is 
conducive to speeding up development of new models for new products DUT's that 
may be of the same technology family. Models are reusable in the sense that if the 
behaviour exhibited by the model or a component of the model is consistent with the 

20 behaviour intended by a new or different implementation of a DUT for which the 
original model was designed, then it is possible to re-apply elements of the model 
directly or after some minor amendment As noted above, in accordance with a 
prefenred embodiment, state based beha^riou^al models are created In a modelling 
environment 101 such as Autofbcus^. A handcrafted or user defined 100 Test 

25 Configuration File (TCF) is parsed through the model preferably in a compiled C 
version which is an executable giving rise to a Test Data Fomiat (TDF) file which 
represents the input test vectors generated based on the information confined in 
the TCF and which may generally correspond to a TVF. This TDF of input only 
stimuli (test vectors) which comprises the same Input set of stimuli as the golden set 

30 is generated and parsed, time stamped, filtered and formatted and the resultant TDF 
from this process is utilised to drive 104 an automation test interface 106 developed, 
preferably in a 1-abview™ environment. Alternatively other moduies could provide 
the same functionality that Labview^^ provides, nainnely a programming environment 

14 
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per se, for example, an ADA test environment. Labvi^"^^ conveniently has 
associated with Its programming environment, off the shelf I/O solutions that 
comprise drivers that facilitate direct integration vfilOn the Labviow™ applications. It 
is. however, possible to develop a similar solution in ADA, C. Perl etc. A separate 

5 TDF IS generated, which comprises input vectors (stimuli) to output vectors 
(responses) generated by the model based on the behaviour represented by the 
model* The resultant TDF described here is retained as the golden set of test input 
108 to test data against which the output 107 of the DUT 105 will be compared* In a 
preferred embodiment Labview^ VIrihJal Instmments (Vl's) are developed 106 that 

10 are DUT 105 specific retaining knowledge of the unique port names and variables 
utilised In the model and have knowledge of the specific command language of the 
bur 105* Physical interfiaoing to the DUT 105 may be via specific I/O cards. The 
Vrs may be controlled and coordinated by a master VI or test "manager^ 106, The 
<jr master VI has the task of controlling the timing and execution of the tests as 

15 represented in the parsed and filtered TDF, or drive TDF (DTDF) mentioned eariier. 
By reading the Input vectors represented in the DTDF the Vl's generate the 
appropriate electrical stimuli necessary to stimulate the inputs of the device DUT 105 
and to corrfigure it. The outputs of the DUT 105 are subsequently monitored and the 
responses recorded 107. A log or execution TDF (ETDF) In a fonnnat consistent with 

20 the DTDF IS consequently generated through the course of the test execution. Post 
test execution the DTDF and ETDF may be automatically compared 109 via an 
appropriate script or in a l-abvlew^ environment The differences in response from 
the actual DUT output as compared to that of the original TDF from the compiled 
model or oracle 108 are flagged as potential defects or deviations from expectations 

25 based solely on the original product function and performance requirements. 
Parsing of a TDF Pile 

Below i$ an example of an output TDF file after an input only TDF has been 
passed through a model. The prefenned format of the TDF betow is 
<{nput port >?<pattem>; ... <output port>!<pattem>; 
30 in this example Ports 1«*3 are input ports and Ports 4*6 are output ports. 

Portl?x;Port2?y;Port3?z?Port4 1 /Ports i ;Poft6 ! ; 
■ • ■ 

35 Portl?3c;pQrt:2?y;Port37ZfPort;4la;Port;5! jPortSl ; 

15 
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Portl?x;Port2?y;Port3?z;Port4!a;Port5!b;Port6 I ; 

repeated n number of times as the input vectors are passed through the 

S model 

Portl?xyPcifti2?yrPox-fc3?z;£>ort4 layPortS Jb;Port:6 £ ; 

until eventual^ all of the output ports (represented by <portname>I<valiie>) 
10 are populated. 

Port 1 7x ; Port2 ?y ; Ports ? z ; Port4 l a ; Poart 5 5 b ; Port 6 ! c ; 
Portl?x?Port2?y;Port3?z;Port4 ia.;Port5 ib;Port6 1 c? 
Poart 1 7x ? Port2 ?y f Port3 ? z : Por t4 i a $ Ports ! to ; Port 6 ! c ; 

15 

however, even at the point of populeting aii output ports, the values contained 
theiBin may change In the time It takes to dock through the entire modeK This may 
vary dependant upon how the model was implemented, that is, variation may be 
dependent on the nature of the device being modelled. For example, a smoke 
20 detector will not immediately produce a tme output value at the time a stimulus is 
provMed at Its input However there will come a time when the output stops 
changing becoming a stabilised output and It Is this vector that must be used to 
model and test the device. 

25* Portl?x ; Port 2 ?y ; Port 3 ? z ; Port 4 i a ; For 1 5 I b ? Port 6 1 c ; 

PcrtlVx ; Port2 ?y ; Ports ? z ; Port4 ! a ; Ports I to ; Port 6 2 c; 

Port 1 ?x ; Por t2 ?y ; Port 3 ? z ; For t4 ! a ; Ports 1 b ? Ports ! c ; 

Portl?X;Port2?y;Port3?z;Port4 la;Port5 lb;Port6 ic; 

Porti?x ; Port2 ?y ? Port3 ? z ? Port4 1 a ; Port:S l b $ Ports I c ; 
30 Portl?x? Porta 7y; Port3 ?a ; Port4 1 a? Port 5 lb; Port 6 Ic ; 

Eventually a point in time arrives when the outputs have stabilised and a 
matdhing input and output vector (a vector pair) may be attained. The Issue is at 
what point In time after the input stimulus Is applied will the output be valid? A 
35 number of different methods have been employed by the inventor as follows: 

1 . Wait a fixed period of time - this may prove to be quite inefficient and 
lilies on the fact that each test vector pair doesn't rely on what state the last vector 
pair left the system In. In a woridng system this was flawed because It didn't allow 
the model designer - the tme test engineer - to create sequence's (Inter linked 
40 vector pairs) 

16 
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2. Wait until the actual outputs satisfy a predefined match of the vector 
pair. This Is also flawed because it requires waiting until the output matc;hes. 
However, it is usual to employ some kind of time out In ih\^ situation otherwise if the 
outputs never match, there will be no prognsssinQ through the remainder of the 

5 vector pairs. Employing a timeout has the same symptoms as those in 1 above. 

3. Count the numt>er of lines within a TDF before the Test Vector pair and 
use that to determine the time before the outputs are valid. This is may not be valid 
since the number of test vectors will vary based upon the implementation of the 
model, two models of the same DUT may be implemented in different ways that 

10 will require a different number of dock pulses to generate the same test vector pair. 

4. l^ave the test vector Indicate a predestined period of time before If s 
output vector is valid* This is in essence achieved by creating extra timing ports, for 
example, in the model. The timing ports Indicate to the automated execution engine 
what period of time to wart before sampling the outputs of the system under test. 

15 This underlies a preferred emtx>diment of the present invention. 

Generic Automafed Test Execution 

It Is desirable to implement an execution engine in a generic fashion that 
allows the re-use of the same architecUire across multiple products and projects, 
DUT's. This requires a test execution engine that doesnt have any knowledge of 
20 the DUT or the interface executing the test 

Executing automated tests Involves a number of steps, 

Configure DUT 

Start test on OUT 

Implement a system *Wair for a period time until outputs can be verified 
25 IVIeasure output values 

Evaluate results against the expected values* 

Generic lExeGUtlon Engine Architecture 

The architecture of the Genetic Execution Engine (GEE) 20 in accordance 
with a preferred embodiment is shown in Fig 2. This architecture 20 supports either 
30 multiples of the same DUT 23 to fadlrtate parallel lest execution or different DUT's 
23 being tested simultaneously* 



17 
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A number of advantages are provided by the GEE 20 firstly, it allows for 
accelerated life tests, greater test coverage in the same period of time, independent 
testing of multiple DUT's or mulUpIo instances of the same DUT. A further 
advantage is in terms of reliability, ie should one DUT fall It is independent of all 
5 other DUrs and tests being executed on them. 

The architecture for the GEE 20 detail* a number of interfaces 

(i) TDF-GEE Interfiaice 

(ji) GEE-DUT Interface 

(iii) GEE-ResuRLoglnterfeoe 

10 TDP'^BE Interface (I) 

The TDF-OEE Interfeee resides between the GEE 20 and model generated 
TDF files 21 . This Interface provides the function detailed above In relation to the 
parsing of a TDF file. 

GEE-DUT Infeerface (fi) 
15 This is a communication Interface utilising TCP/IP In a prefeffed embodiment. 

However, any other Interprocess communication method may be employed such as, 
for eoomple. Acthra-X comprising tor example. DOOM or COM. The soclcet address ^ 
for a given DUT 23 is identified by a TCP/IPJKddress port, which is preferably 

contained in a TDF 21 . 
20 The GEE 20 manages the GEE-DUT Interface and provides the following 

services to DUT specific lntorfBce(8) 22. 

(i) Test Vector Pair - provide the test vector pair to the DUT specific 
Interfoce 22 

(ii) Restart Test -Re-send test vectors storting from the first test vector 

25 pair in the TDF 21 

(III) Add result to log 24 - DUT specific Interface 22 provides test vector 
pair and resultant vector pair for processing by GEE to provide formatted actual data 
to results log 24 
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GEE-R08Utt Log Interface (iii) 

The format of the Result Log 24 is an ascii format file. The file name used 
may be generated by concatenating together the value of the DUT 23 port in the 
TDF 21 and a time and date stamp. 
5 The file contains data formatted in the following way. 

• DateTimeStamp. 

• Evaluated result +++ =PASS. 

++. = PASS (masked port failure encountered) 
~ = FAIL. 

^0 *** B Enor encountered during test 

« Binary String representing a logical comparison of each port value 
between the Test Vector Pair and Resultant Vedsr I^lr. The user is able to mask 
the comparison of any ports that th^ choose. 

• Each port Is listed and if the data ircahdies only one value la shown. 
15 Othenwise, If values don't match then the ojq^ected value is presented first foltewed 

by the actual value. 

Examples of each sequence R^ults Log file described above are shown in 
figures Sa to 3c, respectively. 
Further elaboratkin of the present vivention follows. 
20 Test Generation 

The inventor defines a test as: 

"A set of preconditions, stimuli, and expected responses or outcomes so 
an^inged as to achieve the objective of exercising a function or behaviour to verify 
compliance with a requirement specification". 

25 in a preferred embodiment our tmpiementation ensures that a measure of 

control is maintained in the test generation and this may be achieved by opting to 
cmft the values chosen. It may be understood that Judicious selection of 
equivalence class partitions and inclusion of ad product/device presete and 
preconditions is an exeralse that tmy provide a meaningful ^t of parameters to 

30 generate the stimuli. The parameters Identmed are employed in the combinatorial 
test vector generatton in order to taiter the HavouT of the vectore generated. This is 
done in onler to ensure that spedfto state transittons or particular state machines 
are covered in very exf^toit fashton under directed control. The reason for this is that 

19 
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in many instances there may be a timo owJer ImpUcit in the behaviour we wish to 
test 

Onoe the test vectors are generated we may ensure that groupings are . 
created that provide for specific modes of execution. It is deslralale that during 
automatic execution to ensure that certain events have oocuned in a particular order 
for registers to have been set and local variables to have been updated for tests to 
have significance or meaning. 

The result of the automatic test vector generation activity is a set of stimuli we 
call the input Test Vector RIes, as described above. 

To generate the actual teste involves passing the input Test Vector Files 
(TVF's) through the model that possesses dynamic behaviour and capturing the 
output which typically is a paired Input/Output Test Vector File. 

The automatic test generation circle Is closed at this point. The effective 
result is that the model becomes an orade or predictor of the product behaviour, 
1 5 direcUy providing the "expectation" element of the "tesT deflnitton above. 
The Patti To Automatic Execution 

To achieve automatic execution it is considered that this requires that once 
the model has been mn under simulation and the output TVF generated, the input 
TVF is parsed through the automation infrastmcture of the present invention. The 
20 actual physical product under test is subsequently stimulated by the same sel: of test 
. vectors as the model although now employing a test harness to provide the electrical 
and mechanical intertaces to the real world environment. The product responses 
are captured and logged. These product responses are compared to those 
produced by the model eartlar and any deviations are noted for Investigation. 
25 Metrics 

The chosen metrics show that apprcMimately as much time Is spent 
developing the model as would be spent in developing manual teste for the same 
funotfonalfty. The difference is that in return virtiat we gain from the Investment in 

building the models is: 
30 « Sup^rtor review capability of the originating requirements 

• simulation capability to investigate dynamlcally the requirement 
interaction 

* Capabilify to generate phenomenal numbers of tests 
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m A means to provide test vectors as a result of the latter that can be 
exploited by automation Infrastructure 

« Ease of maintenance in terms of rapid reaction to requirements 
change, models are re-usable and easily amended. 

m As models are easily amended so too the tests generated through 
those models are as easHy and quIcWy regenerated to account for the changes 

Additional Benefits 

Employing a modeling tool constrains the acttvlty to a highly structured and 
fonnal approach and the rigour of the approach forces the test engineer to critically 
analyse the requirements under consideration. This rigor as a consequence 
provides for highly effective review. In Itself this formal review through the modeling 
approach has proven useful in identifying inaccuracies, Inconsistencies and 
contradictions as well as conflicts in the original requirements documents. 

In addition, the experience of the Inventor has proven that actually Interacting 
in a dynamic sense with a model provides additional benefits in as much as the 
potentially unwanted behaviour arising from strict adherence to the requirements can 
be, and has in many Instances been, Identified and exposed early enough to be 
corrected before actual implementation, in other words the test engineer through 
dynamic modeling creates a very eariy prototype of the product elements they are 
Interested in and Interacts with them in such a manner as to understand, appreciate 
and expose the nuances of the resultant behaviour. 
Automatic Test Execution 

. Executing the test 

The automated test execution engine reads the TVF generated by the 
modeling environment as discussed eartier. The TVF contains the test vectors to be 
executed by the execution engine. In this fHe an entire test description ts contained 
on one line. 

<input port >?<pattem>; ... <output port>!<pattem>: 

This single line gives a complete definition for a test vector. It provides us 
with a complete set of configuration parameters and stimulus to drive the test. It 
also provides us with a complete list of expected responses^ which we then compare 
with the actual resulte to evaluate the result 

21 
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As previously noted 9 limitation with the above format on its own Is that it 
doesnt convey any timing inforniation. To solve this the preferred embodiment of 
the invention provides timing ports, which the modeling environment uses to convey 
5 timing infomtatron to the execution engine. These timing ports don't represent 
physical ports on the DUT, but are virtual ports used apecificalty by the Execution 
Engine to schedule timed events during the execution of a test vector. 
We have defined the following port types in our implementation: 

10 • Configuration Port -This defines configuration parameters that map to 

physical parameters/inputs on the DUT. 

• Timing Port - Used to schedule the timed events during the execution of a 
testvecfenr. 

• Stimulus Port - This port is physically mapped to an input on the DUT. In 
15 our case this port defines what stimulus Is to be applied to initiate the test. 

• Output Port - These map to physical outputs on the DUT. For example 
these could be relay contacts. LED's, Ul Display message, serial message or an 
analogue output. 

20 The execution of a test l8.detalled in the flow chart of Table 1 showing the 

Execution Engine STD. The STD highlights the steps rsquirsd at a macro level to 
execute a test vecton 

1 . Read Test Vector to obtain the entire ConfHJuration. Stimulus. Timing and 
25 Output parameter information. 

2. Conf^uie DUT with configuration parameters identified by the 

Configuration Porta. 

3. Stimulate DUT witti stimulus identified by the Stimulus Ports. 

4. Wailftor DUT timing event. Sometimes referred to as a testing hook into 
30 flie system. |n our cai» this is a data message returned from the device to indicate 

that it has pixtcessed the stimulus and begun the timing algorithm. 

5. Wait duration indicated by Timing Port. This is used to Indicate the time at 
which theoutputs should be in tiie conect state. 

22 
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6, Sampla Outputs. This Involves recording the actual Inputs that were 
applied, performing serial communications with the DUT and reading discrete 

outputs from the DUT* 

7. Log Results, store the results into a file for later analyste- 

8, tog Period parameters. Some parameters are logged on a periodic rate 
to assist with the evaluation of the test results. 

9. Sample Period, The sample period Is usually detemnined by the dynamic 

nature of the parameter being logged. 



Begin autDznafeed execution 



IJlead 



DXJT 






3^tbniilate 
DXJT 



DUT 
Tixiimg«v«D!t 
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8. Periodically 
log«elected 



9-Wait 
Pciiodic Saniple 
time 



5,Wait 
Duration 
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outputs 



7.Log 
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Table 1 - Excctttion Engine STD 
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Analysing The Results 

The inventor originally envisaged that analysis would be the simpler activity, 
however, experience had shown that It may be the most time consuming activity, 
due to the textual nature of the Log FUes. To assist with the analysis and 
5 significantly reduce the analysis time an embodiment of the invention provides a 
"LogViewer". 

The Lo^iewer allows the perfomfiance of a number of repetitive functions 
quickly and easily. It provides the ability to: 

10 * Scroll badovaitl and fonvanls b^ween individual test vectors 

* Scroll backwaid and Ibnivards between failed test vectors 

* Dynamically change the evaluation criteria by masking out particular 
ports from the evaluation 

* Generate a list of failure types based on the evaluation criteria 
15 • Synchronise ail results wKh the current test vector, including aU 

periodically togged data that had been captured during e)»cution 

* Graphically summarize the ports that had different data values 

lastween expeded and actual results. 

m Scroll through adjacent test vectors to provide the developers with a 
20 good picture of the events that led up to failure. 

* Ability to handle large numk>ers of test vectors and large sets of 
periodic data. 

The execution engine captures periodic data and graphs It over time whilst 
25 also scheduling the capture of output ports in accordance with the timing dictated by 
the model. Both of these sets of data are stored for post analysis and evaluation. 

Multiple test vectors may often display the symptoms of a single failure, The 
ability to diareicterizse these symptoms into failure types helps to group faBurss 
30 automatically. Providing a list of the fa'dure types based upon the parameter mask 
(described above) applied allows the test engineer to focus their efforts. 
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Failures are grouped Into types by creating a logicaf comparison of each 
port's actual versus expected result These logical comparisons are placed into a 
binary string and like binary strings may be characterized as a failure type. 

The LogViewer also allows filtering out or masking of certain ports. It may be 

5 helpfiit when filtering out test vectors that show symptoms related to particular ports- 
Accuracy and Error Tolerance - Whenever the modeled wortd is compared 
with the real worid this issue may be encountered. There are many ways to address 
this ieeue, some being highly complex and some being quite simple. Whatever 
method chosen to employ will depend greatly on the problem being solved, in the 

10 case of the present invention a simple solution is presented. In the case of an 

embodimont of the present Invention we evaluate the acceptability of a result within 
the LogViewer The expected results may be compared vAth the actual and have a 
tolerance, which the test engineer can dynamically change. This allows the test 
engineer to investigate accuracy by varying the tolerance band and observing the 

15 change in failure rate. 

Given the ease with which an automatic test generation engine can 
combinatorially generate vast numbers of test vectors from modest sets of 
parameters even small increases to the original parameter sets will result in 
exponential increases in the effective numt>er of vectors produced. 

20 The questton needs to be then asked, what is adequate? 1 000 test vectors^ 

lO.OOOtestvectors. 100.000 vectors, 1,000,000? IVIore? 100 Billion? These 
numbers may be possible. It Is advisable to consider carefully the equivalence 
classes and boundary oondittone; ensure that great care is taken to minimize the 
effectual ranges employed and the granularity utillilsed to generate the equivalence 

25 classes. Combinatorial generation carries the problem of state space explosion if 
not carefully managed. Nevertheless, the present inventioh has provided for the 
generation of test vector numbers in excess of 50,000. The overall execution time 
for tliese vectors is dependent on the real time behavtour of the functionality that Is 
being exeroi$ed. Where the product is a communications protocol, for example, the 

30 time to execute each test vector may be In the order of milliseconds to seconds, 
conversely where the behaviour Is related to opening or closing valves this could be 
seconds to minutes. It may depend in the product under teat, however given that the 
intent is that the test vectors are executed automatically then vast numbers can be 
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so executed over night or awoss week-ends which means the cost of execution is 
reduced. 

Model Coverage and Requirement Coverage Issues 
One of the benefits of modeling with a CASE tool Is that it may be possible to 
map what portions of the model have In feot been exorcised by exploitation of the 
generated input test vectore. This is useftil in that H indicates for the values chosen 
for each of the parameters being exercised what the resultant model coverage Is. 
Given that the model as a minimum is based on the product requirements under teat 
and mosHy the transitions and states that reflect the requirements arise directly from 
the requirement statements a fair assumption that 100% model coverage may 
equate vtrtth 100% requirement coverage. 

While this Invention has been described in connection with specific 
embodiments thereof, it will be understood that It is capable of further 
modlflcation(s). This application is intended to cover any variations uses or 
adaptations of the Invention following in general, the principles of the invention and 
including such departures from the present disclosure as come within known or 
customary practice within the art to which the invention pertains and as may be 
applied to the essential features hereinbefore set forth. 

As the present invention may be embodied in several fonns without departing 
from the spirit of the essentia! characteristics of the Invention, it should be 
understood that the above described embodiments are not to limit the present 
invention unless otherwise spedfled, but rather should be constraed broadly within 
the spirit and scope of the invention as described. Various modlfioatfons and 
equrvaient aitangements are Intended to be included within the spirit and scope of 
the invention. Therefore, the specific embodiments are to be understood to be 
Ulusti^e of the many ways In Whioh the principles of the present Invention may be 
pracHced. In statements of the invention herein, means-plus-function clauses are 
Intended to cover stmctuies as perfonning the defined function and not only 
stmcturai equivalents, but also equivalent stnjcturcs. For example, although a nail 
and a screw may not be stnjctural equivalents in that a nail employs a cylindrical 
surface to secure wooden parts together, whereas a screw employs a helical surface 
to secure wooden parts together, in the environment of fastening wooden parts, a 
nail and a screw are equivalent structures. 
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-Ccwnprises/comprfslng" when used in this specification Is taken to specify the 
presence of stated features. Integers, steps or components but does not preclude 
the presence or addition of one or more other features, integers, steps, components 
or groups thereof." 

5 

DATED 7 July 2004 

Vision Fire jS^ Security Ply Ltd 

By DAVIES COLUSON CAVE 
Patent Attorneys for the applicant 

10 
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ABSTRACT 

The present Invention relates to the fieW of embedded or distributed systems. 
In one aspect, the present Invention provides in an embedded DUT testing system 
comprising apparatus adapted to compare actual DUT Inputfoutput vector pairs wHh 
modeoed DUT inputtoutput vector pairs, a logical connection port adapted to Indicate 
a predefined flmmg reference for detenninlng a point in time at which to sample an 
output vector as the conespondlng output vector in an Input/output vector pair, 

in another aspect the present Invention provides for the testing of at least one 

embedded DUT comprising: 

a) detenninlng a lest configuration parameter set comprising predefined DUT 

test sequence rules; 

b) determining a first data set comprising input test vectors based on the test 

configuration parameter set; 

c) processing the first data set In a DUT model to determine output test 
vectors wherein the output test vectors comprise DUT model generated responses 
to the Input te$t vectors; 

d) processing the first data set and the output test vectors to delennine a 
second data set comprising pairs of stabilised input and output test vectors; 

e) communicating the stabilised Input test vectors to at least one DLJT via a 
DUT independent interface so that the at least one DUT is stimulated by the 
stabilised Input test vectors to produce DUT output vectors; 

f) detennining a third data set comprising the stablHsed input vectors and 
conespondlng DUT output vectors; 

g) comparing the third data set with the second data set to determine a 
comparison of actual behaviour to modelled behaviour of the.at least one DUT. 

Figl 
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